The present invention relates to correcting distortion in images by computer, and to scanning wide fields of view by moving a lens over a detections system.
An electronic camera uses a conventional lens but in place of film employs an array of electronic light sensor elements such as CCD""s (charge coupled devices). Typically the array is rectangular. Signals from the respective sensors may be converted directly into an image on a display, for example a corresponding rectangular array of LED""s (light emitting diodes). The sensor signals may also be stored in a computer memory, or processed by a computer CPU.
Conventional cameras using electronic pickup arrays are known in which a small sensor array is movable about the image plane. Such cameras cover a wider field of view, with a given lens, than would otherwise be possible. One example is described by Topps in U.S. Pat. No. 5,206,503. The sensor array, smaller than the usable focal plane area of the lens, is movable within the image plane in two directions, denoted X and Y. If the sensor array is coupled to a TV, the image translates as the sensor undergoes X-Y motion, even when the lens is stationary.
Very similar to Topps is U.S. Pat. No. 4,928,174 to Smith. Smith discloses a surveillance camera providing a narrow field of view, i.e. a relatively high magnification, with a lens focussing on a relatively small CCD array for video pickup. The CCD array is movable, within the focal plane, to change the view. FIGS. 6a-6c of Smith show the basic idea. The movable video pickup (CCD array) provides a mechanical improvement over the more conventional method of moving the entire camera body around because the sensor array is smaller, better protected, etc.
Although it is not of concern to Smith, the images from Smith""s movable-pickup camera are not exactly the same as those provided by the more conventional swivelling-body camera in which the lens, and its optical axis, change direction. This is because of perspective.
When an amateur photographer takes a picture of a tall building, invariably the camera is tilted up: both the optical axis of the lens and the film at an angle. On the resulting photograph the building is in the shape of a triangle, with its parallel sides inclined on the paper. (The same effect is seen in a photograph taken of railroad tracks when the camera lens optical axis is parallel to the tracks: they converge to the horizon on the photographic print, even though they are of course actually parallel.)
In contrast, an architectural photographer may take a picture of the same building, from the same vantage point, but using a view camera having adjustments not available on other cameras. These adjustments allow the lens""s optical axis to remain horizontal and permit the film to move downward while the film plane remains perpendicular to the lens axis. In the resulting photograph the sides of the building appear as they are, parallel.
The more conventional surveillance camera, in which the axis tilts, provides a perspective like that of the nonadjustable camera, while Smith""s movable-pickup camera mimics the professional view camera and provides a different perspective.
Smith does not comment on this perspective difference, and for simple surveillance purposes with moderate lenses the perspective is not crucial. However, in the case of an extreme-wide angle lens and a pickup movable to positions far off the optical axis, the distortion produced would become noticeable. Photographs taken with extreme wide-angle lenses, e.g. a 15-mm lens on a 35-mm camera, appear distorted even when the lens actually has no xe2x80x9cdistortionxe2x80x9d in the sense used by opticians or lens designers. (An optician considers a lens to distort when the rays from objects to the corresponding points on the image plane does not all pass through a single point inside the lens, that is, when the angle of the image point off the optical axis is the same as the angle of the object point off the optical axis. Such distortion causes a straight object to have a curved image. xe2x80x9cPin-cushionxe2x80x9d and xe2x80x9cbarrelxe2x80x9d distortions are examples.)
Moreover, even if the lens is optically perfect (without opticians"" distortion) and the angle is not extreme, the images from different lens positions do not correlate. For example, suppose a person is photographing a mountain scene with a camera having a horizontal field of view of 30xc2x0. She takes a picture with the camera pointed north, then another with the camera pointed 30xc2x0 to the east, then another pointed 60xc2x0 to the east, and so on. She prints the pictures and then constructs a panorama by aligning the edges of the pictures. It will be found that the photograph images do not correlate correctly at the photograph borders.
To be more precise, there will be a point-to-point matchup at the border between two photographs, so that a line crossing the border (e.g. a telephone wire) will meet there. But the angles of the line in the two photos will be different: objects which are actually straight will appear to bend at the border. This effect will be referred to as xe2x80x9cangular distortionxe2x80x9d.
It will be found that angular distortion becomes worse as the field of view increases If the photographer in the example above had used a camera with a 45xc2x0 field of view and shoot pictures at 45xc2x0 increments, the distortion would be worse. If a telephoto lens having a narrow field were used, the effect would be negligible and the photographs could be juxtaposed without noticeable angular distortion.
Because of this effect, panoramic cameras confine the film to one vertical strip at the image plane and rotate the camera to provide a continuous change in the azimuthal angle of the lens optical axis. This avoids the sharp bends in the images of such objects as roof lines which occur with panoramas made of pasted-together photographs. In a panoramic picture the image of a straight object such as a roof line often appears as a curved line on the photograph.
The origin of angular distortion is two-fold.
First, the outer portions of an image are magnified as compared to the inner portions, because the film is farther from the optical center of the lens. Referring to FIG. 1, the optical center C of the lens L is the point inside the lens which corresponds to the position of a pin-hole lens producing the same image. The distance from the lens center C to the plane of the sensor-element pixel array SP is clearly greater when the image points are away from the optical axis A.
As in a telephoto lens, which has a long focal length, the image at the corners of the image plane (coinciding with the pixel array SP) is enlarged because the effective focal length of the lens L is greater at the corners than in the center.
Consider the two point objects OP separated by a small angle xcex1. The distance between their image points on the sensor array SP, denoted as image pair IP, is a function of xcex8. When the two close-set points OP are both near to the optical axis their separation on the film is almost exactly equal to the focal length of the lens times the angular separation between them, measured in radians; but when offset from the axis as illustrated in FIG. 1 their two image points IP will be separated by a greater amount, namely the on-axis distance divided by the cosine of xcex8.
In other words, the magnification factor is 1/(cos xcex8). Because the function cosine xcex8 is approximately constant and equal to one in the neighborhood of xcex8=0xc2x0, there is little angular distortion there. But as the off-angle increases, so does distortion.
The second origin of angular distortion has to do with the tilt of the focal or film plane with increasing xcex8. The distance between the image pair of points IP will move farther apart from one another as xcex8 increases (while the angular separation xcex1 of the object pair OP stays constant), but will move apart still faster if the line between them passes through the optical axis, as compared to the case where the line between them is perpendicular to the optical axis. This is because the focal plane is tilted away from the optical axis. In FIG. 1 the dashed line between the pair of image points IP shows that they are in a line extending from the optical axis A.
FIG. 2 shows the extra separation in the radial direction. If the surface of the sensor pixel array SP were tilted perpendicular to the incoming image rays, the distance between would be d; but on account of the tilt their separation is e. Trigonometry gives e=d/(cos xcex8).
Prior-art electronic cameras have not taken these distortions caused by perspective into account, nor corrected them. In particular, they have provided no way to correlate images from one portion of an image plane with images from another portion. This has prevented the use of electronic cameras to form high-quality composite images.
In particular, prior-art electronic cameras have not electronically displaced the image points in memory as any function of perspective. If a prior-art sensor array were 500 by 500, the data from the pixels has typically been put into a 500 by 500 memory array and/or plotted on a plotter having a 500 by 500 dot matrix The resulting perspective is just like that of a conventional film camera, and images cannot be juxtaposed.
Similarly, prior-art electronic cameras have not provided any way for correcting opticians"" distortions, which can also prevent images from juxtaposing with adjacent images properly.
Accordingly, the present invention has an object, among others, to overcome deficiencies in the prior art such as noted above. A particular object is to provide composite or built-up images from a plurality of sub-images, without distortion (including angle distortion or any other distortion caused by the optical element in front of the pickup sensor, or due to inaccurate placement of the sensor elements).
The invention provides a method for removing angle distortion by adjusting raw sensor data using trigonometric functions, and/or other mathematical treatment, especially as to X-Y location. This permits building up composite images without any distortion.
Making this correction on data derived from a flat image plane will remove angular distortion entirely and mimic the effect of a spherical image plane. Since film comes in sheets and is held flat in a camera, lenses have been designed to focus onto a plane surface, resulting in angular distortion. Some specialized cameras, notably the Schmidt camera, use a spherical film surface and a special lens. A xe2x80x9cpinholexe2x80x9d camera also can use a spherical film surface (indeed, a surface of any shape at all). Clearly, the spherical photographs from a Schmidt camera would be fastened at their edges to form a globe constituting an image of the camera""s entire surroundings, extending 360xc2x0 in every direction and totally without distortion (assuming no optician""s distortions). The present invention mimics this complete juxtaposability electronically, without requiring unusual lenses or curved sensor arrays.
In the present invention signals from the detector pixels are not simply placed directly into a memory array corresponding point-by-point with the pixel array of the pickup; instead, the signals are trigonometrically adjusted, preferably via a simple computer program, preferably so that there is zero angular distortion. The factors of cosine xcex8 are removed mathematically to reduce or eliminate angle distortion, permitting images to be overlapped at will without discontinuities.
It will be apparent that correction of angular distortion involves multiplying the distance between two image points IP by a factor of (cos xcex8) if the line joining them is tangent to a circle centered on the optical axis, and multiplying the distance between two image points IP by (cos2 xcex8) if the line between them passes through the optical axis A.
In the case of the flat focal-plane lens L, flat image plane, and rectangular sensor array SP of FIG. 1, the correction is made by finding the radial distance r of each outlying sensor pixel P1 from the point P0 where the optical axis meets the pixel array SP. The coordinate are of course arbitrary, but conveniently may be originated at point P0, which is assigned coordinates (0, 0) while point P1 has coordinates x1, y1, point P2 has coordinates x2, y2, etc.
The correction for any two close-lying image points P1 and P2 is as follows: The radial distance r of the neighborhood of P1 and P2 is determined. This distance could be the distance of either one, the mean average of their radial distances, etc. From r the angle xcex8 is easily found; by geometry, xcex8=arctangent (r/f), where f is the focal length of the lens. To correct for the magnification effect the distance between the two points, i.e.
[(x2xe2x88x92x1)2+(y2xe2x88x92y1)2]xc2xd,
is multiplied by cos xcex8.
The position of any pixel element or point P in the plane of the sensor array SP cannot be described only in terms of r; two coordinates are needed. Circular coordinates, r and xcfx86, are most natural. In these r is again the radial distance and xcfx86 is the angle measured from some line lying on the plane of the sensor array SP. The distance along the direction of the angle xcfx86, denoted as the tangential distance, is equal to rxcfx86 for close-set points. Of course, rectangular coordinates x and y are easily converted to circular coordinates, and conversely back again, by standard formulas.
The initial multiplication by the factor (cos xcex8) has already accounted for the magnification, but an extra correction in the r direction is still needed because the radial distance varies as cos2 xcex8, not cos xcex8.
To complete the correction the difference in radial distance between the points P1 and P2, denoted xcex94r, is found as a first step. This difference is multiplied by cos xcex8, so that the radial separation between the points is now corrected by a factor of cos2 xcex8, while the tangential separation is corrected by a factor of cos xcex8. This completes the transformation from a planar image with angle distortion to an undistorted image which can be plotted directly onto a spherical, rather than a flat, surface.
Angular distortion is only one type of distortion, and with the present invention any other lens distortions can be removed by computer means if the distortion is geometrical and its function is known.
In the general case the distortion is given by a function of the following form:
F=g(x,y),
where x and y are the coordinates of the point in the lens image plane, g is a general function, and F is the amount of distortion. (F need not be a vector quantity; due to radial symmetry of the lens, the direction of distortion can be taken as radial from the center point at the intersection of the optical axis and the image plane.)
In the case of a moving lens the x and y values of a certain pixel on the sensor surface are calculated using the following formulae:
xxe2x80x2=X+x,
yxe2x80x2=y+y,
where X and Y are the magnitude of movement generated by the moving element and (x,y) are the pixel coordinates on the CCD surface in respect to the center point.
In the present invention the mathematical transformation corresponds to a map projection in which points are plotted onto the plane of the map by straight rays from the center of the globe which rests on the plane, the rays passing through points on the globe to the corresponding points on the map. (Since the invention transforms the other way, it is more precisely the inverse of this map transformation.)
Because in the present invention the data is all mapped onto a virtual sphere, the present invention will allow the juxtaposition of data sets obtained from two cameras pointed different ways, or from one camera set with its optical axis pointing in two different directions sequentially. The entire surroundings of the camera, round 360xc2x0, can be stored in memory.
It is well known to map makers that a sheet of paper cannot cover a portion of a sphere without folding, and this is the reason for distortion of area, shape, or angle in maps of the globe. For this reason the virtual map of the invention can only be printed out or televised exactly onto a spherical surface. However, any portion of the stored virtual spherical image can be plotted, printed, or displayed on a flat surface merely be reversing the algorithms used to generate it.
Thus, a virtual globe or star map could be generated in memory with conventional lenses and video pickup sensor arrays, using only a small number of cameras or a small number of exposures with one camera.
Furthermore, a small movable sensor array can be used, in the manner of Smith, to generate an image without angle distortion. This reduces the cost of the array, and does not harm the image quality since the virtual data picture can be built up while scanning.
Still further, the data from two of more cameras with movable sensor arrays (or alternatively a single-pixel sensor element) can be correlated so that one single virtual image is generated.
Preferably, the lens is movable for simpler wiring although a movable detector is also adaptable to the present invention. Except when the objects to be imaged are very close to the lens, there is little difference in the resulting images between a camera with a movable lens and one with a movable sensor array.
The present invention corrects for any distortion at all, including opticians"" distortions in which straight objects have curved images. Therefore the invention permits recording of high-quality images with the use of less-expensive lenses, as well as juxtaposing several such images without border mis-matches.
The present invention can also correct for intentional or designed-in distortions. For example, an extreme wide-angle xe2x80x9cfish-eyexe2x80x9d lens, covering an angle of 180xc2x0, produces a round image rather than a rectangular image, and introduces severe distortion. The horizon, which of course appears straight to the naked eye, becomes an enclosing circle on the fish-eye image plane when the optical axis is aimed at the zenith, and is strongly curved whenever the axis is not horizontal. The present invention can transform such a distorted image into an un-distorted, virtual hemispherical image from which any portion can be transformed back into a plane picture, but without distortion.
The present invention can act as a xe2x80x9cvirtual view cameraxe2x80x9d, changing the apparent perspective from one plane image to a second plane image with software corresponding to the independent tilts and swings of a view-camera""s lens holder and film holder.